2021-11-19  c0638991ef5cf95b421aac211d60e7ef 99+   29 分钟   4.4 k0次访问

关于 Kubernetes中etcd的一些笔记

一个人思虑太多，就会失去做人的乐趣。——莎士比亚

写在前面

  • 学k8s涉及到一些etcd的知识，所以学习后这里系统整理下。
  • 博文内容涉及：
    • 单节点etcd搭建，版本切换
    • etcd集群搭建:用两个节点初始化，然后动态添加一个节点到集群。
    • etcd集群数据快照恢复
    • 为了方便，使用ansible，所以阅读本文需要知道一点ansible

一个人思虑太多，就会失去做人的乐趣。——莎士比亚

一、etcd概述

etcd是CoreOS团队于2013年6月发起的开源项目，它的目标是构建一个高可用的分布式键值(key-value)数据库。etcd内部采用raft协议作为一致性算法，etcd基于Go语言实现。类似于微服务项目中服务注册与发现组件： Eureka、 Zookeeper 和 Consul 等。

etcd具有以下特点：
完全复制：集群中的每个节点都可以使用完整的存档
高可用性：Etcd可用于避免硬件的单点故障或网络问题
一致性：每次读取都会返回跨多主机的最新写入
简单：包括一个定义良好、面向用户的API(gRPC)
安全：实现了带有可选的客户端证书身份验证的自动化TLS
快速：每秒10000次写入的基准速度
可靠：使用Raft算法实现了强一致、高可用的服务存储目录

二、单节点ETCD搭建

机器：192.168.26.91

安装etcd包

1
2
3
4
5
┌──[root@liruilongs.github.io]-[~]
└─$ yum -y install etcd
┌──[root@liruilongs.github.io]-[~]
└─$ rpm -qc etcd
/etc/etcd/etcd.conf

修改配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
┌──[root@liruilongs.github.io]-[~]
└─$ vim $(rpm -qc etcd)
┌──[root@liruilongs.github.io]-[~]
└─$
#[Member]
# 数据位置
ETCD_DATA_DIR="/var/lib/etcd/default.etcd" 
# 数据同步端口
ETCD_LISTEN_PEER_URLS="http://192.168.26.91:2380,http://localhost:2380" 
# 读写端口
ETCD_LISTEN_CLIENT_URLS="http://192.168.26.91:2379,http://localhost:2379" 
# etcd名字
ETCD_NAME="default" 
#[Clustering]
ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"

设置开机自启

1
2
3
4
5
6
7
8
9
┌──[root@liruilongs.github.io]-[~]
└─$ systemctl enable etcd --now
┌──[root@liruilongs.github.io]-[~]
└─$ etcdctl member list #查看数据源  
8e9e05c52164694d: name=default peerURLs=http://localhost:2380 clientURLs=http://localhost:2379 isLeader=true
┌──[root@liruilongs.github.io]-[~]
└─$ etcdctl cluster-health #查看数据量
member 8e9e05c52164694d is healthy: got healthy result from http://localhost:2379
cluster is healthy

isLeader=true 是否为leader

基本命令测试

1
2
3
4
5
6
7
8
9
10
11
12
13
┌──[root@liruilongs.github.io]-[~]
└─$ etcdctl ls /
┌──[root@liruilongs.github.io]-[~]
└─$ etcdctl mkdir cka
┌──[root@liruilongs.github.io]-[~]
└─$ etcdctl ls /
/cka
┌──[root@liruilongs.github.io]-[~]
└─$ etcdctl rmdir /cka
┌──[root@liruilongs.github.io]-[~]
└─$ etcdctl ls /
┌──[root@liruilongs.github.io]-[~]
└─$

2和3版本切换

1
2
3
4
5
6
7
8
9
10
11
12
┌──[root@liruilongs.github.io]-[~]
└─$ etcdctl -v
etcdctl version: 3.3.11
API version: 2
┌──[root@liruilongs.github.io]-[~]
└─$ export ETCDCTL_API=3
┌──[root@liruilongs.github.io]-[~]
└─$ etcdctl version
etcdctl version: 3.3.11
API version: 3.3
┌──[root@liruilongs.github.io]-[~]
└─$

三、etcd集群构建

  • ETCD集群是一个分布式系统,使用Raft协议来维护集群内各个节点状态的一致性。
  • 主机状态 Leader, Follower, Candidate
  • 当集群初始化时候，每个节点都是Follower角色，通过心跳与其他节点同步数据
  • 通过Follower读取数据，通过Leader写入数据
  • 当Follower在一定时间内没有收到来自主节点的心跳，会将自己角色改变为Candidate，并发起一次选主投票
  • 配置etcd集群，建议尽可能是奇数个节点，而不要偶数个节点,推荐的数量为 3、5 或者 7 个节点构成一个集群。

1、创建集群

这里使用ansible的方式实现

vms81.liruilongs.github.io 控制节点
192.168.26.100 etcd节点 受控节点
192.168.26.102 etcd节点 受控节点
192.168.26.101 etcd节点 受控节点

环境准备，主机清单编写

1
2
3
4
5
6
7
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$cat inventory
......
[etcd]
192.168.26.100
192.168.26.101
192.168.26.102

受控节点ping测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -m ping
192.168.26.100 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}
192.168.26.102 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}
192.168.26.101 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -m yum -a "name=etcd state=installed"

配置文件修改

这里用前两台(192.168.26.100,192.168.26.101)初始化集群，第三台(192.168.26.102 )以添加的方式加入集群

本机编写配置文件。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$cat etcd.conf
ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"

ETCD_LISTEN_PEER_URLS="http://192.168.26.100:2380,http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.26.100:2379,http://localhost:2379"

ETCD_NAME="etcd-100"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.26.100:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379,http://192.168.26.100:2379"

ETCD_INITIAL_CLUSTER="etcd-100=http://192.168.26.100:2380,etcd-101=http://192.168.26.101:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
参数的意义
ETCD_NAME 节点名称，默认为defaul
ETCD_DATA_DIR 服务运行数据保存的路
ETCD_LISTEN_PEER_URLS 监听的同伴通信的地址，比如http://ip:2380，如果有多个，使用逗号分隔。需要所有节点都能够访问，所以不要使用 localhost
ETCD_LISTEN_CLIENT_URLS 监听的客户端服务地
ETCD_ADVERTISE_CLIENT_URLS 对外公告的该节点客户端监听地址，这个值会告诉集群中其他节点
ETCD_INITIAL_ADVERTISE_PEER_URLS 对外公告的该节点同伴监听地址，这个值会告诉集群中其他节
ETCD_INITIAL_CLUSTER 集群中所有节点的信息，格式
ETCD_INITIAL_CLUSTER_STATE 新建集群的时候，这个值为 new；假如加入已经存在的集群，这个值为existing
ETCD_INITIAL_CLUSTER_TOKEN 集群的ID，多个集群的时候，每个集群的ID必须保持唯一

把配置文件拷贝到192.168.26.100,192.168.26.101 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.100,192.168.26.101 -m copy -a "src=./etcd.conf dest=/etc/etcd/etcd.conf force=yes"
192.168.26.101 | CHANGED => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": true,
    "checksum": "bae3b8bc6636bf7304cce647b7068aa45ced859b",
    "dest": "/etc/etcd/etcd.conf",
    "gid": 0,
    "group": "root",
    "md5sum": "5f2a3fbe27515f85b7f9ed42a206c2a6",
    "mode": "0644",
    "owner": "root",
    "size": 533,
    "src": "/root/.ansible/tmp/ansible-tmp-1633800905.88-59602-39965601417441/source",
    "state": "file",
    "uid": 0
}
192.168.26.100 | CHANGED => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": true,
    "checksum": "bae3b8bc6636bf7304cce647b7068aa45ced859b",
    "dest": "/etc/etcd/etcd.conf",
    "gid": 0,
    "group": "root",
    "md5sum": "5f2a3fbe27515f85b7f9ed42a206c2a6",
    "mode": "0644",
    "owner": "root",
    "size": 533,
    "src": "/root/.ansible/tmp/ansible-tmp-1633800905.9-59600-209338664801782/source",
    "state": "file",
    "uid": 0
}

检查配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.100,192.168.26.101 -m shell -a "cat /etc/etcd/etcd.conf"
192.168.26.101 | CHANGED | rc=0 >>
ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"

ETCD_LISTEN_PEER_URLS="http://192.168.26.100:2380,http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.26.100:2379,http://localhost:2379"

ETCD_NAME="etcd-100"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.26.100:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379,http://192.168.26.100:2379"

ETCD_INITIAL_CLUSTER="etcd-100=http://192.168.26.100:2380,etcd-101=http://192.168.26.101:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
192.168.26.100 | CHANGED | rc=0 >>
ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"

ETCD_LISTEN_PEER_URLS="http://192.168.26.100:2380,http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.26.100:2379,http://localhost:2379"

ETCD_NAME="etcd-100"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.26.100:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379,http://192.168.26.100:2379"

ETCD_INITIAL_CLUSTER="etcd-100=http://192.168.26.100:2380,etcd-101=http://192.168.26.101:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

修改101的配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.101  -m shell -a "sed -i  '1,9s/100/101/g' /etc/etcd/etcd.conf"
192.168.26.101 | CHANGED | rc=0 >>

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.100,192.168.26.101 -m shell -a "cat -n /etc/etcd/etcd.conf"
192.168.26.100 | CHANGED | rc=0 >>
     1  ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"
     2
     3  ETCD_LISTEN_PEER_URLS="http://192.168.26.100:2380,http://localhost:2380"
     4  ETCD_LISTEN_CLIENT_URLS="http://192.168.26.100:2379,http://localhost:2379"
     5
     6  ETCD_NAME="etcd-100"
     7  ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.26.100:2380"
     8
     9  ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379,http://192.168.26.100:2379"
    10
    11  ETCD_INITIAL_CLUSTER="etcd-100=http://192.168.26.100:2380,etcd-101=http://192.168.26.101:2380"
    12  ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
    13  ETCD_INITIAL_CLUSTER_STATE="new"
192.168.26.101 | CHANGED | rc=0 >>
     1  ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"
     2
     3  ETCD_LISTEN_PEER_URLS="http://192.168.26.101:2380,http://localhost:2380"
     4  ETCD_LISTEN_CLIENT_URLS="http://192.168.26.101:2379,http://localhost:2379"
     5
     6  ETCD_NAME="etcd-101"
     7  ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.26.101:2380"
     8
     9  ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379,http://192.168.26.101:2379"
    10
    11  ETCD_INITIAL_CLUSTER="etcd-100=http://192.168.26.100:2380,etcd-101=http://192.168.26.101:2380"
    12  ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
    13  ETCD_INITIAL_CLUSTER_STATE="new"
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

查看etcd集群

192.168.26.101 为 Leader

1
2
3
4
5
6
7
8
9
10
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.100,192.168.26.101 -m shell -a "etcdctl member list"
192.168.26.100 | CHANGED | rc=0 >>
6f2038a018db1103: name=etcd-100 peerURLs=http://192.168.26.100:2380 clientURLs=http://192.168.26.100:2379,http://localhost:2379 isLeader=false
bd330576bb637f25: name=etcd-101 peerURLs=http://192.168.26.101:2380 clientURLs=http://192.168.26.101:2379,http://localhost:2379 isLeader=true
192.168.26.101 | CHANGED | rc=0 >>
6f2038a018db1103: name=etcd-100 peerURLs=http://192.168.26.100:2380 clientURLs=http://192.168.26.100:2379,http://localhost:2379 isLeader=false
bd330576bb637f25: name=etcd-101 peerURLs=http://192.168.26.101:2380 clientURLs=http://192.168.26.101:2379,http://localhost:2379 isLeader=true
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

添加etcd 192.168.26.102到集群 

1
2
3
4
5
6
7
8
9
10
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.100 -m shell -a "etcdctl member add etcd-102 http://192.168.26.102:2380"
192.168.26.100 | CHANGED | rc=0 >>
Added member named etcd-102 with ID 2fd4f9ba70a04579 to cluster

ETCD_NAME="etcd-102"
ETCD_INITIAL_CLUSTER="etcd-102=http://192.168.26.102:2380,etcd-100=http://192.168.26.100:2380,etcd-101=http://192.168.26.101:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

修改之前写好的配置文件给102

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$sed -i '1,8s/100/102/g' etcd.conf
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$sed -i '13s/new/existing/'  etcd.conf
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$sed -i 's#ETCD_INITIAL_CLUSTER="#ETCD_INITIAL_CLUSTER="etcd-102=http://192.168.26.102:2380,#' etcd.conf
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$cat -n etcd.conf
     1  ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"
     2
     3  ETCD_LISTEN_PEER_URLS="http://192.168.26.102:2380,http://localhost:2380"
     4  ETCD_LISTEN_CLIENT_URLS="http://192.168.26.102:2379,http://localhost:2379"
     5
     6  ETCD_NAME="etcd-102"
     7  ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.26.102:2380"
     8
     9  ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379,http://192.168.26.102:2379"
    10
    11  ETCD_INITIAL_CLUSTER="etcd-102=http://192.168.26.102:2380,etcd-100=http://192.168.26.100:2380,etcd-101=http://192.168.26.101:2380"
    12  ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
    13  ETCD_INITIAL_CLUSTER_STATE="existing"
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

配置文件拷贝替换，启动etcd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.102 -m copy -a "src=./etcd.conf dest=/etc/etcd/etcd.conf force=yes"
192.168.26.102 | CHANGED => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": true,
    "checksum": "2d8fa163150e32da563f5e591134b38cc356d237",
    "dest": "/etc/etcd/etcd.conf",
    "gid": 0,
    "group": "root",
    "md5sum": "389c2850d434478e2d4d57a7798196de",
    "mode": "0644",
    "owner": "root",
    "size": 574,
    "src": "/root/.ansible/tmp/ansible-tmp-1633803533.57-102177-227527368141930/source",
    "state": "file",
    "uid": 0
}
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.102 -m shell -a "systemctl enable etcd --now"
192.168.26.102 | CHANGED | rc=0 >>
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

检查集群是否添加成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -m shell -a "etcdctl member list"
192.168.26.101 | CHANGED | rc=0 >>
2fd4f9ba70a04579: name=etcd-102 peerURLs=http://192.168.26.102:2380 clientURLs=http://192.168.26.102:2379,http://localhost:2379 isLeader=false
6f2038a018db1103: name=etcd-100 peerURLs=http://192.168.26.100:2380 clientURLs=http://192.168.26.100:2379,http://localhost:2379 isLeader=false
bd330576bb637f25: name=etcd-101 peerURLs=http://192.168.26.101:2380 clientURLs=http://192.168.26.101:2379,http://localhost:2379 isLeader=true
192.168.26.102 | CHANGED | rc=0 >>
2fd4f9ba70a04579: name=etcd-102 peerURLs=http://192.168.26.102:2380 clientURLs=http://192.168.26.102:2379,http://localhost:2379 isLeader=false
6f2038a018db1103: name=etcd-100 peerURLs=http://192.168.26.100:2380 clientURLs=http://192.168.26.100:2379,http://localhost:2379 isLeader=false
bd330576bb637f25: name=etcd-101 peerURLs=http://192.168.26.101:2380 clientURLs=http://192.168.26.101:2379,http://localhost:2379 isLeader=true
192.168.26.100 | CHANGED | rc=0 >>
2fd4f9ba70a04579: name=etcd-102 peerURLs=http://192.168.26.102:2380 clientURLs=http://192.168.26.102:2379,http://localhost:2379 isLeader=false
6f2038a018db1103: name=etcd-100 peerURLs=http://192.168.26.100:2380 clientURLs=http://192.168.26.100:2379,http://localhost:2379 isLeader=false
bd330576bb637f25: name=etcd-101 peerURLs=http://192.168.26.101:2380 clientURLs=http://192.168.26.101:2379,http://localhost:2379 isLeader=true
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

设置环境变量，切换版本，这里有一点麻烦。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -m shell -a "echo 'export ETCDCTL_API=3' >> ~/.bashrc"
192.168.26.100 | CHANGED | rc=0 >>

192.168.26.102 | CHANGED | rc=0 >>

192.168.26.101 | CHANGED | rc=0 >>

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -m shell -a "cat ~/.bashrc"
192.168.26.100 | CHANGED | rc=0 >>
# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
export ETCDCTL_API=3
192.168.26.102 | CHANGED | rc=0 >>
# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
export ETCDCTL_API=3
192.168.26.101 | CHANGED | rc=0 >>
# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
export ETCDCTL_API=3
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -m shell -a "etcdctl version"
192.168.26.100 | CHANGED | rc=0 >>
etcdctl version: 3.3.11
API version: 3.3
192.168.26.102 | CHANGED | rc=0 >>
etcdctl version: 3.3.11
API version: 3.3
192.168.26.101 | CHANGED | rc=0 >>
etcdctl version: 3.3.11
API version: 3.3
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

同步性测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$# 同步性测试
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.100 -a  "etcdctl put name liruilong"
192.168.26.100 | CHANGED | rc=0 >>
OK
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -a "etcdctl get name"
192.168.26.100 | CHANGED | rc=0 >>
name
liruilong
192.168.26.101 | CHANGED | rc=0 >>
name
liruilong
192.168.26.102 | CHANGED | rc=0 >>
name
liruilong
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

四、etcd集群备份，恢复

准备数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.100 -a  "etcdctl put name liruilong"
192.168.26.100 | CHANGED | rc=0 >>
OK
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -a "etcdctl get name"
192.168.26.102 | CHANGED | rc=0 >>
name
liruilong
192.168.26.100 | CHANGED | rc=0 >>
name
liruilong
192.168.26.101 | CHANGED | rc=0 >>
name
liruilong

在任何一台主机上对 etcd 做快照

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#在任何一台主机上对 etcd 做快照
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.101 -a "etcdctl snapshot save snap20211010.db"
192.168.26.101 | CHANGED | rc=0 >>
Snapshot saved at snap20211010.db
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$# 此快照里包含了刚刚写的数据 name=liruilong，然后把快照文件到所有节点
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.101 -a "scp /root/snap20211010.db root@192.168.26.100:/root/"
192.168.26.101 | CHANGED | rc=0 >>

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.101 -a "scp /root/snap20211010.db root@192.168.26.102:/root/"
192.168.26.101 | CHANGED | rc=0 >>

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

清空数据

1
2
3
4
5
6
7
8
9
10
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -a "etcdctl del name"
192.168.26.101 | CHANGED | rc=0 >>
1
192.168.26.102 | CHANGED | rc=0 >>
0
192.168.26.100 | CHANGED | rc=0 >>
0
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

在所有节点上关闭 etcd，并删除/var/lib/etcd/里所有数据：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$# 在所有节点上关闭 etcd，并删除/var/lib/etcd/里所有数据：
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -a "systemctl stop etcd"
192.168.26.100 | CHANGED | rc=0 >>

192.168.26.102 | CHANGED | rc=0 >>

192.168.26.101 | CHANGED | rc=0 >>

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -m shell -a "rm -rf /var/lib/etcd/*"
[WARNING]: Consider using the file module with state=absent rather than running 'rm'.  If you need to
use command because file is insufficient you can add 'warn: false' to this command task or set
'command_warnings=False' in ansible.cfg to get rid of this message.
192.168.26.102 | CHANGED | rc=0 >>

192.168.26.100 | CHANGED | rc=0 >>

192.168.26.101 | CHANGED | rc=0 >>

在所有节点上把快照文件的所有者和所属组设置为 etcd：

1
2
3
4
5
6
7
8
9
10
11
12
13
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -a "chown etcd.etcd /root/snap20211010.db"
[WARNING]: Consider using the file module with owner rather than running 'chown'.  If you need to use
command because file is insufficient you can add 'warn: false' to this command task or set
'command_warnings=False' in ansible.cfg to get rid of this message.
192.168.26.100 | CHANGED | rc=0 >>

192.168.26.102 | CHANGED | rc=0 >>

192.168.26.101 | CHANGED | rc=0 >>

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$# 在每台节点上开始恢复数据：

在每台节点上开始恢复数据：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.100 -m script  -a "./snapshot_restore.sh"
192.168.26.100 | CHANGED => {
    "changed": true,
    "rc": 0,
    "stderr": "Shared connection to 192.168.26.100 closed.\r\n",
    "stderr_lines": [
        "Shared connection to 192.168.26.100 closed."
    ],
    "stdout": "2021-10-10 12:14:30.726021 I | etcdserver/membership: added member 6f2038a018db1103 [http://192.168.26.100:2380] to cluster af623437f584d792\r\n2021-10-10 12:14:30.726234 I | etcdserver/membership: added member bd330576bb637f25 [http://192.168.26.101:2380] to cluster af623437f584d792\r\n",
    "stdout_lines": [
        "2021-10-10 12:14:30.726021 I | etcdserver/membership: added member 6f2038a018db1103 [http://192.168.26.100:2380] to cluster af623437f584d792",
        "2021-10-10 12:14:30.726234 I | etcdserver/membership: added member bd330576bb637f25 [http://192.168.26.101:2380] to cluster af623437f584d792"
    ]
}
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$cat -n ./snapshot_restore.sh
     1  #!/bin/bash
     2
     3  # 每台节点恢复镜像
     4
     5  etcdctl snapshot restore /root/snap20211010.db \
     6  --name etcd-100 \
     7  --initial-advertise-peer-urls="http://192.168.26.100:2380" \
     8  --initial-cluster="etcd-100=http://192.168.26.100:2380,etcd-101=http://192.168.26.101:2380" \
     9  --data-dir="/var/lib/etcd/cluster.etcd"
    10
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$sed '6,7s/100/101/g' ./snapshot_restore.sh
#!/bin/bash

# 每台节点恢复镜像

etcdctl snapshot restore /root/snap20211010.db \
--name etcd-101 \
--initial-advertise-peer-urls="http://192.168.26.101:2380" \
--initial-cluster="etcd-100=http://192.168.26.100:2380,etcd-101=http://192.168.26.101:2380" \
--data-dir="/var/lib/etcd/cluster.etcd"

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$sed -i '6,7s/100/101/g' ./snapshot_restore.sh
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$cat ./snapshot_restore.sh
#!/bin/bash

# 每台节点恢复镜像

etcdctl snapshot restore /root/snap20211010.db \
--name etcd-101 \
--initial-advertise-peer-urls="http://192.168.26.101:2380" \
--initial-cluster="etcd-100=http://192.168.26.100:2380,etcd-101=http://192.168.26.101:2380" \
--data-dir="/var/lib/etcd/cluster.etcd"

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.101 -m script  -a "./snapshot_restore.sh"
192.168.26.101 | CHANGED => {
    "changed": true,
    "rc": 0,
    "stderr": "Shared connection to 192.168.26.101 closed.\r\n",
    "stderr_lines": [
        "Shared connection to 192.168.26.101 closed."
    ],
    "stdout": "2021-10-10 12:20:26.032754 I | etcdserver/membership: added member 6f2038a018db1103 [http://192.168.26.100:2380] to cluster af623437f584d792\r\n2021-10-10 12:20:26.032930 I | etcdserver/membership: added member bd330576bb637f25 [http://192.168.26.101:2380] to cluster af623437f584d792\r\n",
    "stdout_lines": [
        "2021-10-10 12:20:26.032754 I | etcdserver/membership: added member 6f2038a018db1103 [http://192.168.26.100:2380] to cluster af623437f584d792",
        "2021-10-10 12:20:26.032930 I | etcdserver/membership: added member bd330576bb637f25 [http://192.168.26.101:2380] to cluster af623437f584d792"
    ]
}
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

所有节点把/var/lib/etcd 及里面内容的所有者和所属组改为 etcd：v然后分别启动 etcd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -a "chown -R etcd.etcd /var/lib/etcd/"
[WARNING]: Consider using the file module with owner rather than running 'chown'.  If you need to use
command because file is insufficient you can add 'warn: false' to this command task or set
'command_warnings=False' in ansible.cfg to get rid of this message.
192.168.26.100 | CHANGED | rc=0 >>

192.168.26.101 | CHANGED | rc=0 >>

192.168.26.102 | CHANGED | rc=0 >>

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -a "systemctl start etcd"
192.168.26.102 | FAILED | rc=1 >>
Job for etcd.service failed because the control process exited with error code. See "systemctl status etcd.service" and "journalctl -xe" for details.non-zero return code
192.168.26.101 | CHANGED | rc=0 >>

192.168.26.100 | CHANGED | rc=0 >>

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

把剩下的节点添加进集群

1
2
3
4
5
6
7
8
9
# etcdctl member add etcd_name –peer-urls=”https://peerURLs”
[root@vms100 cluster.etcd]# etcdctl member add etcd-102 --peer-urls="http://192.168.26.102:2380"
Member fbd8a96cbf1c004d added to cluster af623437f584d792

ETCD_NAME="etcd-102"
ETCD_INITIAL_CLUSTER="etcd-100=http://192.168.26.100:2380,etcd-101=http://192.168.26.101:2380,etcd-102=http://192.168.26.102:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.26.102:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
[root@vms100 cluster.etcd]#

测试恢复结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.102 -m copy -a "src=./etcd.conf dest=/etc/etcd/etcd.conf force=yes"
192.168.26.102 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "checksum": "2d8fa163150e32da563f5e591134b38cc356d237",
    "dest": "/etc/etcd/etcd.conf",
    "gid": 0,
    "group": "root",
    "mode": "0644",
    "owner": "root",
    "path": "/etc/etcd/etcd.conf",
    "size": 574,
    "state": "file",
    "uid": 0
}
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.102 -m shell -a "systemctl enable etcd --now"
192.168.26.102 | CHANGED | rc=0 >>

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -m shell -a "etcdctl member list"
192.168.26.101 | CHANGED | rc=0 >>
6f2038a018db1103, started, etcd-100, http://192.168.26.100:2380, http://192.168.26.100:2379,http://localhost:2379
bd330576bb637f25, started, etcd-101, http://192.168.26.101:2380, http://192.168.26.101:2379,http://localhost:2379
fbd8a96cbf1c004d, started, etcd-102, http://192.168.26.102:2380, http://192.168.26.102:2379,http://localhost:2379
192.168.26.100 | CHANGED | rc=0 >>
6f2038a018db1103, started, etcd-100, http://192.168.26.100:2380, http://192.168.26.100:2379,http://localhost:2379
bd330576bb637f25, started, etcd-101, http://192.168.26.101:2380, http://192.168.26.101:2379,http://localhost:2379
fbd8a96cbf1c004d, started, etcd-102, http://192.168.26.102:2380, http://192.168.26.102:2379,http://localhost:2379
192.168.26.102 | CHANGED | rc=0 >>
6f2038a018db1103, started, etcd-100, http://192.168.26.100:2380, http://192.168.26.100:2379,http://localhost:2379
bd330576bb637f25, started, etcd-101, http://192.168.26.101:2380, http://192.168.26.101:2379,http://localhost:2379
fbd8a96cbf1c004d, started, etcd-102, http://192.168.26.102:2380, http://192.168.26.102:2379,http://localhost:2379
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$ansible etcd -a "etcdctl get name"
192.168.26.102 | CHANGED | rc=0 >>
name
liruilong
192.168.26.101 | CHANGED | rc=0 >>
name
liruilong
192.168.26.100 | CHANGED | rc=0 >>
name
liruilong
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

k8s中etcd以pod的方式设置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
┌──[root@vms81.liruilongs.github.io]-[~]
└─$kubectl  get pods
NAME                                                 READY   STATUS        RESTARTS   AGE
calico-kube-controllers-78d6f96c7b-79xx4             1/1     Running       57         3d1h
calico-node-ntm7v                                    1/1     Running       16         3d11h
calico-node-skzjp                                    0/1     Running       42         3d11h
calico-node-v7pj5                                    1/1     Running       9          3d11h
coredns-545d6fc579-9h2z4                             1/1     Running       9          3d1h
coredns-545d6fc579-xgn8x                             1/1     Running       10         3d1h
etcd-vms81.liruilongs.github.io                      1/1     Running       8          3d11h
kube-apiserver-vms81.liruilongs.github.io            1/1     Running       20         3d11h
kube-controller-manager-vms81.liruilongs.github.io   1/1     Running       26         3d11h
kube-proxy-rbhgf                                     1/1     Running       4          3d11h
kube-proxy-vm2sf                                     1/1     Running       3          3d11h
kube-proxy-zzbh9                                     1/1     Running       2          3d11h
kube-scheduler-vms81.liruilongs.github.io            1/1     Running       24         3d11h
metrics-server-bcfb98c76-6q5mb                       1/1     Terminating   0          43h
metrics-server-bcfb98c76-9ptf4                       1/1     Terminating   0          27h
metrics-server-bcfb98c76-bbr6n                       0/1     Pending       0          12h
┌──[root@vms81.liruilongs.github.io]-[~]
└─$

关于 Kubernetes中etcd的一些笔记

https://liruilongs.github.io/2021/11/19/K8s/etcd/关于 Kubernetes中etcd的一些笔记/

发布于

2021-11-19

更新于

2023-06-21

许可协议

# 相关文章
  1.ETCD集群通过 Leader 写入数据，为什么K8s HA集群中讲每个 kube-apiserver 只和本机的 ETCD 通信
  2.关于k8s中ETCD集群备份灾难恢复的一些笔记
# 推荐文章
  1.《Kubernetes权威指南:从Docker到Kubernetes实践全接触》读书笔记
  2.Git 提交代码 reference broken 问题处理
  3.关于 Git 重写历史的一些笔记
  4.关于Git分支变基操作的一些笔记
  5.关于Git分支基础知识的一些笔记
  6.关于Git分支工作流的一些笔记

喜欢这篇文章？打赏一下作者吧

评论

目录

关注我

链接

最新评论

加载中，最新评论有1分钟缓存...

最新文章

分类

归档

标签

Kubernetes98
Linux65
Ansible25
Ceph22
Python15
Docker9
Git8
JAVA8
华为云8
BPF7
CPU7
Mysql7
test17
Go6
LLM6
OpenShift6
摄影曝光6
程序人生6
DNS5
OKD5
查看全部>>

订阅更新

Your browser is out-of-date!

Update your browser to view this website correctly.&npsb;Update my browser now

×

×